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ABSTRACT 

The effectiveness of a computer simulation model as 
an alternative means of teaching research methodology to students was 
investigated. Four sections of an introductory course on the subject 
were included in the study. Two were taught through the use of the 
simulation system, while the remaining sections were taught in the 
traditional lecture mode of instruction* The simulation was designed 
to teach skills and concepts in prcMem definition, hypothesis 
generation, sampling, research design, data analysis, interpretation 
and reporting, and the application of those skills and concepts via 
the sittulation problem. Results of the study indicate that the 
simulation approach was very successful in terms of both attitudes 
and skill levels in research methodology. The data suggest that 
students in the experimental groups gained additional skills when 
compared to the traditional sections. However, the lack of a closely 
controlled experimental environment suggests a number of potential 
caveats that should be investigated before further conclusions are 
drawn. (Author/DGC) 
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not threatened by them. However, students in all sections 
felt these concepts were more than a little difficult, 2.8 
on the 7 -point easy-diff icuty scale. 

Fxgure 1 

Average Scores on Six Attitudinal Items 
For All Groups and All Concepts 
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Using only the post test data, collapsing over items 
and ranking the mean scores, the Friedman Two-way Analysis 
of Variance yielded a significant difference among the 
Concepts. Table 1 indicates that groups consistently perceived 
the concepts in the same relative rank order. 

Table 1 

Concept Ranking of Four Areas for the Four Class Sections 



Groups 
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Concepts 


Lecture 
1 2 


Fjcperimental 
3 4 


Educational Research 


1 1 


1 2 


Behavioral Experiments 


2 2 


2.5 3 


Computers 


3 3 


2.5 1 


Statistics 
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1.25 
2.37 
2.37 
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Simulation: An Alternative 
In Teaching Educational Research 



William Loadman Ann Ruedele 
THE OHIO STATE UNIVERSITY 



Universities are constant?- striving to identify viable 
educational alternatives to the traditional lecture format 
utilized by most college professors. With the advent of 
computer hardware in the educational arena, there has been 
increasing interest in its application to the solution of 
educational problems and in the development of new and 
alternative strategies in instructional methods. The 
objective of this study was to ascertain the effectiveness 
of a computer simulation as such an alternative in teaching 
a course in research methodology. 

A >r.aJor problem confronting the use of available hardware 
is the great shortage of and lag in the development of 
accompanying software, i.e., computer programs, and accompanying 
textbooks and materials, to allow adequate experimentation 
with the available hardware. Suppes (1972) and Daley (1971) 
have Clearly indicated this as a major problem area in spite 
of the fact that technical competence, skills and hardware are 
available. However, Collett (1972), of the University of 
Michigan, has developed a computerized simulation of a city 
school district with many of its accompanying problems. One 
of these problems, dealing with the remediation of children 
^ in seventh grade mathematics was operationalized and has been used 
EHjC to teach an introductory course in research methodology. 



Four sections were tested to compare the effectiveness 
of the simulation experience with the traditional lecture 
mode of instruction. Students in this course came from many 
educational programs and were master's or beginning doctoral 
level students. In general, they were naive with respect to 
specific research and/or statistical skills. This course 
was their introduction to the scientific method as a basis 
for inquiry. 

The computer simulation was used to teach two sections 
of the course while two other sections v:ere taught in a 
traditional lecture mode. Students were not aware of any 
differences among the section offerings before enrolling 
except for times when the classes were offered. The instructors 
were not identified in the list of course offerings but one 
instructor taught the two "lecture" sections and another 
instructor taught the two "simulation" sections. 

The simulation was designed to teach skills and concepts 
in problem definition, hypothesis generation, sampling, 
research design, data analysis, interpretation and reporting, 
and the application of these skills and concepts via the 
simulation problem. 

Small working groups were created within the experimental 
classes in order to implement the simulation. Class lecture 
time was reduced by a factor of .5, and the instructor 
utilized the remaining class time to work with the individual 
groups. The instructor acted as an available resource, for 



ERIC 



4 



consultation with the small groups. However^ there were no 
requirements for students to attend lectures or to arrange 
these small group meetings with the instructor. 

Methodology 

Measures of attitudes, skill, knowledge and performance on 
a criterion task were obtained for all four sections of the 
course. The data were analyzed through a series of factorial 
analyses of variance, several one-way analyses of variance, 
and the Walsh Test of differences. Individual scores, as well 
as summated scores, were used as dependent variables. 

Pretest information on attitudes toward computers, 
educational research, statistics and behavioral experiments 
was gathered from all students in the four sectiqns of the 
class. Similar information was collected again at the 
conclusion of the ten-week course. 

In addition, measures of student learning v:ere obtained 
through the use of a series of common items on the four final 
examinations. These common items covered the areas of problem 
definition, hypothesis constmction, sampling, research design, 
statistical analysis, and interpretation of results. 

The experimental groups were also required to prepare 
"journal articles" based on the simulated studies they 
conducted. 

Due to the fact that intact groups v/ere used in this study, 
no causal inferences can be ascribed to the experimental 
manipulation. This limitation must be considered in the 
interpretation of the results of the study. 



Results 

A. Results of Assessment of Attitude Changes 

A pre-/post-course questionnaire concerning perceptions 
of the four basic conceptual areas was administered to students 
in the four sections of the research methodology class. There 
were ^9 students in the experimental sections and 69 students 
in the lecture sections of the course. 

One purpose of this survey was to document student reactions 
in the two experimental sections of the class and to compare 
the affective levels of these students with those of students 
in the two sections taught in the traditional lecture format. 
A second purpose of this survey was to elicit the strength 
of student attitudes, and to note changes over time, toward 
specific methodological concepts basic to this course: 
educational research, behavioral experiments, computers and 
statistics . 

These four concepts were rated on each of six paired 
objective scales: important -unimportant, pleasant-unpleasant, 
valuable-worthless, non threatening-threatening, good-bad, 
and easy-difficult. 

Collapsing the data, over concepts and groups and using 
post-test results, students in all sections considered the 
conceptual areas important, valuable and good. (Ses Figure 1) 
Attitudes were nearly neutral concerning the pleasantness of 
the concepts, and the scores did indicate that students were 
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not threatened by them. However, students in all sections 
felt these concepts were more than a little difficult, 2.8 
on the 7-point easy-diff icuty scale. 

Figure 1 

Average Scores on Six Attitudinal Items 
For All Groups and All Concepts 
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Using only the post test data, collapsing over items 
and ranking the mean scores, the Friedman Two-way Analysis 
of Variance yielded a significant difference among the 
Concepts. Table 1 indicates that groups consistently perceived 
the concepts in the same relative rank order. 

Table 1 

Concept Ranking of Four Areas for the Four Class Sections 



Groups 
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Concepts 
Educational Research 

Behavioral Experiments 

Computers 

Statistics 



Lecture 
1 2 


Experimental 
3 4 


1 1 

2 2 

3 3 

• , , ..tfj 


1 2 
2.5 3 
2.5 1 

1 



R 

1.25 
2.37 
2.37 
4 
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Table 1 (continued) 

'^■^=9.22 p=.013 



Group Ranking of Class Section for Four Concepts 

Groups 



Educational Research 


Lecture 
1 2 


Experimental 
3 h 


3 




1 


2 


Behavioral Experiments 


3 


4 


2 


1 


Compute rs 


k 


3 


2 


1 


Statistics 


3 


4 


2 


1 


R 


3.25 


3.75 


1.75 


1.25 


H 


^ =10.20 


P= 


= .0027 





The ANOVA score obtained for the main effect of Concepts 

wasTC^ = 9.22, p = .013. This indicates that the difference 

in scores among the conceptual areas was statistically 

significant, although all ratings were positive. The 

conceptual area evoking the most positive attitudinal response 

was Educational Research, with Behavioral Experiments close 

behind. Computers were. rated third, and students reported 

their least positive rasponses in the area of Statistics. 

When the Friedman Test was applied to the attitude data 

for the Groups, a signficant difference was uncovered = 10.20 

r * * 

p = .0027). Differences among the four sections in their 
perceptions of the four conceptual areas were significant. 
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The second lecture section consistently evidenced more negative 
attitudes than did the other groups. The two experimental 
sections were nearly equal in attitude strength and were also 
slightly more positive than the first lecture section. 

Using the binomial test it was determined that of the 
16 pre-post differences, I3 of 16 had increased in a positive 
fashion. The probability of this occurring by chance is equal 
to p=.0106. Therefore, it can be con ;luded that in general 
there was a positive shift i.n reported attitudes among the 
students fro^ pre instruction to pest instruction (see Table 2). 

Table 2 

Pre -/Post -Course Mean Scores 
In Four Conceptual Areas* 



Concepts 


Time 


Lecture 
1 2 


Experimental 
1 ? 


Concept 
Means 


Educational 
Research 


Pre-test 
Post-test 


4.99 
5.08 


4.78 
5.03 


5.20 

5o33 


5.17 
5.26 


5.03 
5.17 


Behavioral 
Experiments 


Pre-test 
Post-test 


5.09 
4.99 


4.70 
4.98 


5.08 
5.00 


4.82 
5.25 


4.92 
5.06 


Computers 


Pre-tes t 
Post -test 


5.06 
4.73 


'1.72 
4.84 


4.80 
5.00 


4.94 
5.30 


4.88 
4.97 


Statistics 


Pre-test 
Post-test 


4.56 
4.63 


4.36 
4.48 


4.58 
4.68 


4.76 
4.78 


4.?r 
4.64 


Group Means 


Pre-test 
Post-test 


4.92 
4.85 


4.64 
4.83 


4.91 
5.00 


4.92 
5.15 





* Scale: 7=very positive, 4=neutral, l=very negative 
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Further, analysis of the pre-/post-course attitude changes 
by Means of the Walsh test revealed that in general attitude 
shifts were definitely more positive among the two experimental 
sections than among the two lecture sections (X.^ = 5.8526, 
df = 1, p<.01). Through analysis of the summated raw scores 
for each concept and each group, it was possible to discern 
significant differences in changes in attitudes over time among 
the four groups toward the four basic conceptual areas. 

Using the Walsh Test of differences, the following 
statistically significant differences in attitude changes were 
discovered . 

In the area of educational research , the experimental 
sections began the course with significantly more positive 
attitudes than did the second lecture section. All four groups 
gained in positive attitudes toward educational research by the 
end of the ten-week course, with the second lecture showing the 
greatest net increase. 

The first lecture section and the first experimental section 
began the course with significantly more positive attitudes 
toward behavioral experiments than did the second lecture 
section. However, both the high-scoring groups registered 
declines in positive feelings during the quarter, so that the 
three groups ended the course in research methodology with 
virtually identical overall attitude scores in this conceptual 
area. The second experimental group showed the greatest 
increase in positive affect in this area. 
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The data analysis also supports the conclusion that the 
second experimental section ended the course with significantly 
more positive attitudes toward computers than did either of the 
two lecture sections. 

The second experimental section began the course with 
significantly more positive attitudes toward statistics than 
did the second lecture section. However, the second experimental 
section had the smallest positive increase in attitudes during 
the quarter, and there were no significant differences in 
attitudes among the four groups in this conceptual area at the 
end of the course. 

In summary, use of the computer simulation to teach this 
course in research methodology yielded affective results which 
were equivalent to or more positive than the affective levels 
exhibited by students in the traditional lecture presentation 
of the course with respect to the concepts under study. 
However, all four sections produced positive affect toward 
the four concepts of educational research, behavioral 
experiments, computers and statistics. 
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B. Results of Assessment of Knowledge and Skills 

The basic objectives in all four sections of this research 
methodology course were the same; only the means by which 
instructions were given differed between the lecture and 
simulation sections. In order co assess differences in the 
level of knowledge and/or skill attained ty students in this 
research methodology course, a performance measure was constructed 
and administered at the end of the ten-week course. 

The instructors agreed to I'se a set of 17 common items on 
their examinations, covering the following course objectives: 
Defining the Problem, Research Design, Sampling, Interpretation 
of the Data Analysis and Report Writing. A detailed analysis 
of each of these examination items appears in Table 3. 
(see following page) 
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Table 3 

1 Square Analysis of Common Examination Items 
For Lecture and Fxperlmental Sections 
Of Research Methodology Course 



Objectives 


Lecture 


Experimental 


Item 


Sections 


Sections 


L-E 




f 

c 


% 
c 


f 

c 


c 




Report Writing 


a 


42 


6i 


22 




.21 


Defining the 
Problem 


b 

c 


64 
64 


93 
93 


33 
23 


85 
59 


1.02 

lO mXf 


Sampling 


d 

e 


58 
61 


84 
88 


35 
12 


90 
31 


.28 

37.79** 


Research 


f 

g 
h 
i 

d 
k 
1 
m 
n 
o 

P 


67 
64 

63 
13 
44 
6l 
46 
38 
66 

68 


97 
93 
91 

11 

88 
67 
55 
96 
97 
99 


28 
34 
38 
18 
12 
32 
20 

9 
38 
37 
37 


72 
87 
97 
46 

31 
82 
51 
23 
97 
95 
95 


12.78** 
.38 
.70 
9.08** 
10.82** 

.84 
2.48 
10.45** 

^26 


Interpretation 
of Data 


q 


6o 


87 


31 


79 


1.09 



f^=Frequency of correct response 
^^=Percent of correct response 
**=Differences significant at p<.01 
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The data show that performance on the common Items, 
across all objectives, reflected no significant differences 
between the lecture and simulation sections on 11 of the 17 
items. On five of the Items, the lecture sections were 
significantly superior and on the remaining item the simulation 
sections were significantly superior. In general, one 
experimental section scored consistently, but not significantly, 
higher than the two lecture sections and the other experimental 
section scored consistently lower than the two lecture sections 
(data not shown) . 

A summary analysis of the significance of differences in 
the number of summated correct responses across items on course 
objectives to the common examination items by the experimental 
and lecture sections is shown in Table 4. 

Table 4 

Analysis of Significance of D-' '"erences 
In Number of Correct Responses on Common Exam Questions 
Of Experimental and Lecture Sections, by Course Objectives 



Course Objectives 


Experimental 
Section (N=39) 

^c 


Lecture 

Section (N=69) 

f % 
c c 




Defining the Problem 


28.0 


72 


64.0 


93 


17.17* 


Research Design 


27.5 


71 


5^.3 


79 


9.13* 


Sampling 


23.5 


60 


59.5 


86 


18.76* 


Interpretation of 
Data Analysis 


31.0 


79 


6o.o 


87 


1.09 


Report Writing 


22.0 


56 


42.0 


6l 


.21 


*Chl Square values 
confidence 


are significant at 


the .01 


level of 
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On three of the five course objectives covered by the coiranon 
examination Items, the lecture sections fared significantly 
better than the experimental groups: Sampling, Defining the 
Problem and Research Design. There was no significant 
difference between the two groups Ir the areas of Interpretation 
of Data Analysis and Report Writing. 

In addition to these five objectives, noncoininon examination 
items also covered the areas of: Hypothesis Construction, 
Data Analysis, Validity, Reliability and the Philosophy of 
Science. The data are presented in Table 5. 

Table 5 

Average Number and Percent of Correct Responses 
On Examination Questions of Experimental 
And Lecture Sections, by Course Objectives 
Exp e r imental Lecture 



Course Objectives 



Sections Sections CN=6q) 



Defining the Problem 


29.4 


75 


49.4 


72 


Hypothesis Construction 


28.5 


73 


46.5 


67 


Sampling 


29.0 


74 


40.5 


67 


Research Design 


25.8 


66 


48.2 


70 


Data Analysis 


28.6 


73 


^3.3 


63 


Interpretation of 
Data Analysis 


27.0 


69 


46.2 


67 


Report Writing 


28.5 


73 


49.8 


72 


Assessing Validity 


19.0 


49 


36.0 


52 


Philosophy of Science 


16.0 


4i 


43.0 


62 


Assessing Reliability 






52.7 


76 
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Only the examination for the lecture sections included items on 
tne assessment of reliability. On the nine objectives covered 
by both examinations, the experLnental sections performed better 
on six: Defining the Problem, Hypothesis Construction, Sampling, 
Data Analysis, Interpretation of Data Analysis and Report Writing. 
The lecture sections performed better on the Sampling, Validity 
and Philosophy of Science Items. The differences in performance 
levels between the groups were not significant in any of the 
course objectives. 

It should be emphasized that th° nieans, standard deviations 
and reliabilities of the total exams were very similar across 
sections. Students in all sections displayed a high level of 
understanding of ?nd ability to apply the concepts they learned 
in this course. 

Summary 

Results of the data analysis indicate that the alternative 
to the traditional mode of university teaching met with a great 
deal cf success, reflected not only in student attitudes but in 
the measures of knowledge and skill attainment in the area of 
research methodology. 

Unfortunately, only the experimental sections completed the 
criterion task, i.e., the preparation of a "journal article" 
based upon the computer simulated study. Performance on this 
task was at a very high level. The students demonstrated that 
they could attack a problem, define the parameters and limits 
of the problem, create and implement sampling and research 
designs, select appropriate variables and instruments upon which 
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to collect data, appropriately analyze and Interpret the data 
results and finally communicate their efforts in writing in 
the form of a journal article. The quality of the work of 
these students, who began the course with no research skills 
or experience at the beginning of the quarter, gives some 
evidence of the power of simulation. 

There is ample evidence that simulation is not only a 
viable alternative to the traditional classroom instructional 
mode, but that students in the course spend long hours gaining 
additional skills in research methodology, beyond that of the 
comparison group. 

It should be remembered, in interpreting the results, that 
this study was limited by the lack of control over the research 
situation and by the use of intact groups. Due to the 
limitations, no causal inferences may be ascribed to the 
experimental manipulation. 

Some of the ramifications suggested by this educational 
inquiry are: 

1. Significant amounts of learning occur when the computer 
simulation method of teaching is applied. 

2. There is an increase in the student's positive affect 
toward educational research and evaluation. 

3. The student is able to sit at the computer console and 
generate data, as opposed to the tedious task of actually 
collecting data or generating hypothetical data. 

4. For students in the simulation class, there are many 
real-life research situations. 

5. The cost of simulation is minimal. 
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6. The student proceeds at his own pace, thereby 
reducing many logistic problems. 

7. The simulation method provides an opportunity for 
students to learn from each other as well as from 
the Instructor, 
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